Applying Knowledge from KDD to Recommender Systems

We investigate a new class of software for knowledge discovery in databases (KDD), called recommender systems. Recommender systems apply KDD-like techniques to the problem of making product recommendations during a live customer interaction. These systems are achieving widespread success in E-Commerce today. We extend previously studied KDD models to incorporate customer interaction so these models can be used to describe both traditional KDD and recommender systems. Recommender systems face three key challenges: producing high quality recommendations, performing many recommendations per second for millions of customers and products, and achieving high coverage in the face of data sparsity. One successful recommender system technology is collaborative filtering, which works by matching customer preferences to other customers in making recommendations. Collaborative filtering has been shown to produce high quality recommendations, but the performance degrades with the number of customers and products. New recommender system technologies are needed that can quickly produce high quality recommendations, even for very large-scale problems. For example, traditional KDD techniques might be applied in the context of our model to address these challenges. We have explored one technology called Singular Value Decomposition (SVD) to reduce the dimensionality of recommender system problems. We report an experiment where we use SVD on a recommender system database, and use the relationship between customers in the reduced factor space to generate predictions for products. We observe significant improvement in prediction quality as well as better online performance and improved coverage. Our experience suggests that SVD has the potential to meet many of the challenges of recommender systems.